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Abstract 


Academic reviews take on new directions in an ever-increasing fast-paced development. Unit 
reviewers should be provided different samples, with varied cultural contents also providing 
nuances, as related to the ‘object’ of the evaluation, as well as a final overall product, i.e. 
more of a synthesis, through which one will check the degree of compliance, as regards the 
previously established conditions, and the degree of sensitivity towards the variety of cultures 
inherent in the system. The aim of this paper is to shed light on evolving processes attempting 
to keep up with new contexts. The methodological approach used is qualitative. It consists of 
the analysis of observational notes from the researcher as a participant observer involved in 
the process of two evaluation exercises in higher education. We will present our findings 
using Weiss’ (1980) models for implementation of results in education and discuss underlying 
strategies. Conclusions drawn can be adapted to most educational settings and should enable 
cultural aspects that infiltrate the process to be considered as a value-added characteristic in 
the establishing of a fairer evaluation framework. 
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Introduction 


Evaluation has always favored a certain type of individual and system. For 
others, the results could be influenced by a pre-emptive attitude, as before evaluation 
time all the senses are awakened, and all systems are adjusted to be more efficient 
during the evaluation phase. This sometimes only happens temporarily just before 
the system comes under scrutiny. However, this way of proceeding is contrary to the 
spirit of evaluation. The purpose of monitoring and evaluation is to improve 
individual outcomes, accept responsibilities and adjust vision to aim for the best 
possible progression. 

Considering the importance of outcomes, it is crucial, to also have in mind what 
happens following the data gathering activity. Weiss (1980) described seven models 
whereby the process of implementing research results is carried out in education. 

1. The classic model consists of a follow-up: research, development followed 

by sharing, dissemination and then application. 

2. The problem-solving model presupposes that the researcher presents the 
evidence or conclusions necessary to resolve a problem caused by a 
settlement or the implementation of a regulation. This model is based on 
sequence, identification of the required knowledge, search for appropriate 
knowledge or have it searched by someone, and finally make a decision. 

3. The interactive model is seen as a complex and disorderly interaction 
between researchers and decision-makers. 
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4. The political model presupposes the selective use of search results to support 
or justify a position already taken or to organize a search with the expectation 
that the results will provide confirmations to justify the policy adopted. 

5. The tactical model aims to sweep under the carpet a research problem or to 
avoid the search for a problem to cover the postponement or opposition to an 
action. 

6. The model, called participant research to the company’s intellectual 
enterprise, does not foresee a direct influence of research but seeks the 
impact of the large number of factors that influence various policy factors in 
various ways at different times. 

7. The most common model is to inform the decision-making process through 
an influence on thinking and conceptualization in relation to the questions 
asked. 

The latter is the model Weiss advocates. 

We believe, for our part, that a completely explicit path must be followed, and it 
also seems necessary to define the appropriate terms so that we can qualify the 
procedures, so they are well-understood by all parties involved. 

Beveridge (1996) agrees that there is a tendency to simplify and distort to 
accommodate for simplistic ideas about learning and understanding that the 
education system tends to accept (p. 34). 


Methodology 


Qualitative research is best suited for these two studies in order to find out 
details and uncover intricacies (Creswell & Poth, 2018; Patton, 2014) when looking 
for specifics and making comparisons. The main approach used was an analysis of 
observational notes during evaluation processes at two universities. We are also 
looking at case study as two different situations have been examined in two different 
locations. There is also comparison involved as we are outlining similarities and 
differences in the way the processes unfolded. 

The first study took place before a business model was used for universities. As 
regards the site, we are talking about the internal evaluation of a university institute 
in eastern Canada. The evaluation had to do with a particular academic unit. The 
process lasted three months. Observations were recorded in a journal. Analysis of 
journal texts allowed to uncover emerging themes. These were later discussed in 
light of the approach to the evaluation processes. 

The approach to that evaluation exercise was as follows. The team of reviewers 
consisted of four individuals, including an external reviewer, a highly recognized 
peer in the field of specialty and from the same cultural background as the members 
of the institute. The other three evaluators were on site, one from a similar institute 
within the same university and a culture similar to that of the institute, another of the 
same type of institute at a nearby university and sharing the culture, and the third 
evaluator shared the culture but came from a totally different institute from the same 
university. 

The team members who were on site were tasked with doing a sort of clearing 
ahead of time, consisting of a data collection and summarizing the items to be 
reviewed, from which the external examiner could prepare his two-day investigation 
in the field upon arrival. 
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All members of the institute were summoned to a first meeting during which the 
process was explained in detail and all questions were answered. The secretary of 
the institute also set up schedules to allow each member to have an interview with 
the evaluators. The secretary was also to advise all students at the institute to contact 
the evaluators either for an interview or by sending their comments. After his visit, 
the external reviewer was still allowed to consult the onsite committee through 
phone calls or mail. The committee was also required to submit a full report of the 
process and the issues uncovered as well as recommendations to be made to assist 
the external evaluator in drafting his report. Only the report and recommendations of 
the external evaluator were intended for the university administration and the 
institute. 

The second study was carried out at a university in central Canada of the same 
type as the first one but more recently, particularly when the business model had 
been adopted in academic settings which occurred approximately five years ago. 
The evaluation process observed in this unit did not necessarily reflect what is 
happening elsewhere at the same university and the subjects studied at that unit have 
nothing to do with the business world. Observations were recorded in a journal 
during the evaluation, in the same way as in the first study. These was subsequently 
analyzed, and findings discussed in light of the process for the specific site and later 
comparisons were made between the two processes and the two sites. 

Three knowledgeable evaluators were chosen also perhaps with the hope that 
they would be lenient. The cultural issue however was viewed from a totally 
different perspective. Playmakers recommended their friends as examiners. These 
reviewers therefore shared the culture of these particular members of the institution 
but were probably not particularly representative of the local population. They also 
chose people who shared the same philosophy, no longer looking for neutral 
persons. 

Only management representatives could meet with the reviewers to present all 
the successes of the unit. Members of the institution could not speak to evaluators 
individually, only in small groups organized ahead of time, as if one ‘in-person’ 
were in charge of controlling the discussions. The administration asked to see the 
names of the people who went to these meetings ahead of time, the reason for this 
strategy was unclear. One wonders if someone was infiltrated into the group 
meetings with the reviewers with the mandate to report each person who had a 
complaint to the unit administration. 

A whole productivity file for each member of a group wishing to meet with the 
evaluators had to be submitted to the evaluation committee and prepared 
beforehand: so perhaps the idea was to contextualize the positions of the group 
member who decided to speak-up. All unit members also had to submit their 
productivity records but did not know what the administration was doing with them. 
I requested to have a look at the overall file while meeting with the evaluators in a 
small group meeting and discovered that half of the pages listing my publications 
had been placed after the name of the colleague next to me in alphabetical order and 
who hardly had any publications associated to his name. Headquarters controlled 
this file. 

We had to prepare at least one year in advance of the evaluation. The report of 
the last evaluation was taken up just at that time, with just a year of implementation 
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of the last directions given four years earlier, and it appeared that directions 
suggested during the previous review had received little attention up to that date. 
New processes seemed to have just been implemented as in crisis management. In 
other words, a quick change was made, even in a temporary way to meet the 
requests from previous evaluators. We had to double our effort, mask and embellish 
at every step, in order to influence the result in a positive way. The survival of a unit 
is the concern of every employee. 


Discussion 


The two studies although conducted in different situational contexts showed that 
a great amount of effort was displayed to carry out the evaluation exercises. Despite 
sufficient amounts of care taken, one can wonder how much the results weighed on 
future outcomes. In addition, in light of the vast amounts of data, it would be 
important to know who proceeded with quality control in the organization of files. 
This question concerns the issue of publications data shifted from one colleague to 
another as mentioned above. As a result of the academic review no one ever sent 
detailed comments to unit members, just a one-page summary in each case was 
offered for viewing. 

It is unclear if any negative outcome would have made it to the person 
concemed. There had not been any big issue as a follow-up of the unit evaluations in 
our studies. However, it should be noted that at one point, the second unit received a 
conditional pass only, with a request to submit corrective measures implemented 
that were recommended within a given timeline. Beveridge (1997), and Beveridge 
and Rimmershaw (1991) caution about the complexities involved in analyzing any 
kind of data and unit evaluations that require to put of vast amount of information 
under scrutiny. 


Common trends 


The process in the first study shows a certain openness and consideration for 
cultural diversity. The evaluators were well chosen culturally. They all had 
knowledge of, and an openness to, the culture of the institute. As a result, all 
members of the institute openly contributed, leaving out no details about a problem 
they had at heart. The sharing of the same culture had therefore served to build 
confidence and allow for further evaluation. The committee consisted of two women 
and two men. The young women in the institute who had difficulties to report 
contacted the two women of the evaluation committee. 

One of the problems identified was the gender imbalance at this institute in the 
first study. Another cultural problem that emerged was the differences in conduct 
expected by two different generations. There did not seem however to have been any 
bad intentions in either group, rather a lack of communication. 

It was also discovered that the administration took advantage of the cultural 
differences between members of the institute who were not from the local culture to 
pay them a lower salary. 

However, at the same time there was no complaint about their long absences 
during the summer, thinking that they probably had to return to their country of 
origin to teach a summer session and be better able to provide for themselves. In a 
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sense, this evaluative approach can be considered more humane and humanistic than 
other known models because nothing had been swept under the carpet. However, it 
is mindful of the fact that the only report that is officially recognized is that of the 
external reviewer, who was free to leave out some of the recommendations and 
comments of the local committee. 

All in all, the final report was fair, carried a lot of weight and also respected 
different sensitivities. 

Placing evaluative responsibility on the external reviewer for the final report 
also helped establish a distance between the members of the institute and the local 
examiners, which allowed to maintain good neighborly relations between academics 
sharing the same spaces on campus. These results point to a successful process along 
the lines of Weiss’ (1980) seventh model. 

The second study results came about following observations made during an 
evaluation of a university unit in central Canada, when the business model was 
adopted in academic settings. The choice of evaluators appeared to be judicious 
especially given that more complexity was involved in the business approach to the 
evaluation. The second evaluation was more in line with Weiss’ (1980) second 
model. It also appeared close to a crisis management model. The outcomes of the 
new evaluation were in a sense predetermined by the selection of the evaluation 
committee members. The reviewers made suggestions about new paths proposed, of 
course, but in a superficial and unsustainable way to enable the institute to shine 
despite shortcomings. It was doubtful that this would lead to improvement and the 
same approach to crisis management would probably follow again. 
Recommendations were made, but not directly implemented, probably not until just 
before the next assessment would be scheduled, to show that there has been progress 
made. This approach cannot lead to profound change. There is a lot of food for 
thought stemming from these studies. As student population bases shift drastically 
and at a very fast pace, there undoubtedly needs to be a more concerted effort in 
taking into account the diversity of situations also involving variety and variation 
across a number of spectrums. 


Principles underlying the evaluation process 


One would assume that any evaluation basically would have to do with balance. 
Ways must be found to align all elements of the evaluation with the individuals that 
need to be evaluated. In other words, it is a matter of using the most appropriate 
method in line with contextual information. 

The evaluation exercise must be valid, reliable and fair. It is necessary to see 
what is being done and to what degree of satisfaction if not perfection. This cannot 
be carried out without a calibration system. It is clear that clear expectations and 
explicit criteria for the objectives had to be established beforehand. 

As mentioned above various methods and diversified strategies must come into 
play along evaluation activity divisions into two areas: it is essential to include 
representative samples of the performance of the individuals involved as well as 
copies of behaviors examined on the job. 

A conclusion can only be drawn by taking into account the training streams and 
specific instructions given that are at the source of the behaviors. It has to do with 
culture. 
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The evaluative approach adopted must aim to lead to success, to bring about 
improvement. To that end, assessors and evaluators should make decisions about 
what will be placed under scrutiny together and how, that is, by what criteria, the 
‘objects’ of the evaluation will be assessed. 

This makes a lot of sense and it is understandable that units no longer accept to 
be evaluated without preparation after sufficient notice is given. 

A lot of thought and preparation should go into the process. An awareness for 
evaluators of their own culture is required. 

In the assessment act, the evaluator's culture is very often imposed on the 
‘object’ of the evaluation. In addition, personal ways of using a language also 
condition this act. 

De Gramont (1990) predicts distortions caused by language but which are 
mainly due to lack of cultural knowledge. He tries to identify the problem by 
drawing up a list of metaphors of the self. It includes a hereditary and biological 
component, and other components, including contextual, conventional and poetic. 
These should allow the individual to be well positioned in relation to the assessment 
to be undertaken. 

The hereditary component relates to our physical aspects and the mental traits 
that characterize us. Our biology dictates our behavior to some extent. The 
contextual component refers to the influences of our environment on our 
individuality. This can influence our habits, our working conditions, our responses 
to certain stimuli. Everything about culture, the customs that explain the cultural 
mold that formed us, is found in the conventional component. In the multicultural 
context in which we assess, it is necessary to question the various cultural influences 
surrounding it. Our creativity is placed within the framework of the poetic 
component. The perspective from which a person approaches the issues can also be 
explained by the greater or lesser degree of creativity they possess. 

There should be an openness to the multiplicity of possible cultural clues. In 
addition, higher psychological processes have to be taken into account (Vygotsky, 
1978) when placing institutes of higher education under scrutiny. According to 
Borden (1991), there are a whole range of aspects that influence our assessment of 
situations. The questions he asks relate to eye movement, the importance attached to 
time, positioning, colors, noise and silence, and gestures. 

We need to be aware of such factors because without taking into account the 
traits inherent in cultural habits could lead to distorted interpretations. 


Conclusion 


Knowing how to operate fairly in a unit evaluation is crucial. 

Following the consideration of evaluators’ characteristics of their own cultural 
coloring, in addition to the awareness of other factors that various cultural filters 
have missed, it is also necessary to take stock of the diversity of skills. Assessing 
properly will depend on a given context and on the way the assessment will have to 
change depending on the situation. What could a reasonable operating framework 
look like? For a successful evaluation, it is a matter of working on two aspects. First, 
we must gather facts: we take into account the accumulation of information, 
documents from various sources, whether it be summaries of written activities, oral 
presentations made at meetings, reports of individual productivity or review of 
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finished products, but always in the light of established expectations and not a 
virtual and perfectionist goal. On the contrary, we must take into account the current 
aspect of things, see what can be improved in relation to pre-established 
expectations and choose exemplary ways to demonstrate opportunities for 
improvement. 

It is essential to draw up the list of ‘objects’ to be put under scrutiny at the 
beginning because afterwards, once the process has begun, there will no longer be 
the necessary distance for judgment to remain neutral. 

The tricky part is to choose the items to be evaluated in a culminating way to be 
able to demonstrate that the system holds together soundly. It will be important that 
the small samples selected at the outset reflect the overall aim and add to the value 
of documents that will come under the microscope at an advanced stage, with all of 
it coming together at the time of the synthesis of all the data accumulated. 

On the other hand, it is also necessary to review the ‘collected objects’ and 
judge them on their value, establish their quality and decide on the value to be 
attributed to them. As final remarks, it should be said that since culture infiltrates 
any evaluation process, an accurate picture can only be made if one is prepared to 
take into account the knowledge, know-how and knowing how to be, associated with 
the people whose ‘objects’ are part of the evaluation. 


References 


Beveridge, M. (1996): Strategic Review of Educational Research. Report prepared for the 
Leverhulme Trust. 

Beveridge, M. (1997): Educational implementation and teaching. In L. Smith, J. Dockrell & 
P. Tomlinson (Eds.) Piaget, Vygotsky and beyond (pp.15-26). London: Routledge. 


Beveridge, M. & Rimmershaw, R. (1991): Teaching and tutoring systems; explanatory 
dialogues in context. In P. Goodyear (Ed.) Teaching Knowledge and Intelligent Tutoring 
(pp. 279-296). New Jersey: Ablex. 

Borden, G. (1991): Cultural Orientation: an approach to understanding intercultural 
communication. Englewood Cliffs: Prentice Hall. 

Creswell, J. W. & Poth, C. (2018): Qualitative inquiry and research design. Thousand Oaks, 
Cal.: Sage. 

De Gramont, P. (1990): Language and the Distortion of Meaning. New York: New York 
University Press. 

Patton, M. Q. (2014): Qualitative research and evaluation methods. 4" Ed. Thousand Oaks, 
Cal.: Sage. 


Vygotsky, L. S. (1978): Interaction between Learning and Development. In M. Cole, V. John- 
Steiner, S. Scribner & E. Souberman (Eds.) Mind and Society: The Development of 
Higher Psychological Processes (pp. 79-91). Cambridge, MA: Harvard University Press. 


Weiss, C. H. (1980): Social science research and decision making. New York: Columbia 
University Press. 


Prof. Dr. Marie J. Myers, Queen’s University, Canada 


Educational Reforms Worldwide 


